dual-attention

Awni00's group workspace

Group: Abstractor - L=1, d=128, h=8

1-5

of 5

Tags

Notes

Author

awni00

State

Finished

Start time

August 2nd, 2024 4:16:26 PM

Runtime

5h 9m 25s

Tracked hours

Run path

dual-attention/dual_attention--math--algebra__sequence_next_term/098r9psq

Linux-4.18.0-477.51.1.el8_8.x86_64-x86_64-with-glibc2.28

Python version

3.11.7

Git repository

git clone https://www.github.com/awni00/abstract_transformer

Git state

git checkout -b "expert-hill-9" 9ce16be9a1cd1fba943e52654fc94c965461f2e0

Command

/gpfs/radev/project/lafferty/ma2393/abstract_transformer/experiments/math/train_abstractor_model.py --task algebra__sequence_next_term --n_epochs 100 --batch_size 512 --d_model 128 --dff 256 --symbol_type symbolic_attention --n_layers 1 --n_heads 8

System Hardware

CPU count	32
Logical CPU count	32
GPU count	1
GPU type	NVIDIA A40

W&B CLI Version

0.17.5

Group

Abstractor - L=1, d=128, h=8

Config parameters are your model's inputs. Learn more

▶
Config parameters:{} 12 keys
- ▶
  abstractor_kwargs:{} 12 keys
- d_model:
  128
- ▶
  decoder_kwargs:{} 7 keys
- ▶
  encoder_kwargs:{} 6 keys
- group:
  "Abstractor - L=1, d=128, h=8"
- in_block_size:
  161
- ▶
  input_spec:{} 2 keys
  - type:
    "token"
  - vocab_size:
    85
- n_layers_dec:
  1
- n_layers_enc:
  1
- out_block_size:
  31
- out_dim:
  85
- ▶
  output_spec:{} 2 keys
  - type:
    "token"
  - vocab_size:
    85

Summary metrics are your model's outputs. Learn more

▶
Summary metrics:{} 10 keys
- epoch:
  99
- extrapolate_loss:
  0.06803567707538605
- extrapolate_perplexity:
  1.3182429075241089
- extrapolate_teacher_forcing_acc:
  0.907868504524231
- interpolate_loss:
  0.06803567707538605
- interpolate_perplexity:
  1.3182429075241089
- interpolate_teacher_forcing_acc:
  0.907868504524231
- train_loss:
  0.05778918042778969
- train_teacher_forcing_acc:
  0.8962574601173401
- trainer/global_step:
  390,599

This run produced these artifacts as outputs. Total: 1. Learn more

wandb-history

run-098r9psq-history:v0